ID-45: Session CLI and Scripted Execution#111
Conversation
| **Options**: | ||
| - `-I`, `--inferno_base_url`: URL of the target Inferno service. | ||
|
|
||
| **Output**: JSON array of objects with `name` and `value` keys representing the session's current inputs. |
There was a problem hiding this comment.
THis section seems to be about the session data and the next section is about results, but the hyperlink here is for results and the next is for session data -- they seem flipped
There was a problem hiding this comment.
thanks for catching that.
| - `-s`, `--expected_results_session`: Session ID on the same server. The results of this indicated | ||
| session will be used as the expected results. When the compared session's results do not match | ||
| comparison details will not be written to file (use the `-f` option). | ||
| - `-m`, `--compare_messages`: Also compare per-test messages (not only pass/fail result). Default: `false`. |
There was a problem hiding this comment.
Looking at your most recent PR, I think --compare_messages and --compare_result_message actually default to true now, not false
There was a problem hiding this comment.
They default to true when the execute_script CLI command uses session compare, but not when directly calling the CLI command. So I think this is correct as is.
There was a problem hiding this comment.
| matches: ^There is a problem with the terminology resources | ||
| ``` | ||
|
|
||
| Note that the `normalized_string:` key never appears under the `sessions:` key. |
There was a problem hiding this comment.
normalized_strings (with an s)
| started by this script. The following sub keys are used. | ||
| - `runnable:` *Required* - the test, group, or suite to execute, specified as a short ID | ||
| from the UI or an internal id. | ||
| - `inputs:` *Optional* - key-value pairs indicating inputs to be merged into those already |
There was a problem hiding this comment.
In the comment for inputs in the inferno core repo, you mention two ways to define inputs:
# input_name: "value" # for the input
# input_name: "@path/to/file.txt" # prefix value with @ to read from a file;
# # relative paths are resolved from the
# # directory containing this script file
Should these two methods be listed here?
There was a problem hiding this comment.
yes, and there's actually another possibility - objects or sequences that get serialized to json. Added both here and added the object/sequence option to the core documentation.
|
|
||
| The `--implementation-guide` or `-i` option will look at an absolute file path if its prefixed by `file:///` or lookup the name in the [official FHIR package registry](https://packages.fhir.org/). | ||
|
|
||
| ## Manage Sessions |
There was a problem hiding this comment.
I ran this section through claude to see if it identified any significant gaps in what was defined in this doc vs what options developed have, and it pointed out that there is no documentation at all of the "planned failure" tests, with the _failure naming convention. It details:
_failure script naming convention is undocumented
Neither the scripting doc nor the CI/CD doc mentions that scripts whose filename contains _failure receive special treatment. There are now two pass conditions for a _failure script:
Script exits non-zero and "Actual results matched expected results? true" appears in output.
Script exits non-zero, output contains "errors", and the expected file doesn't exist yet (first-run bootstrap case).
A developer wanting to add a known-bad regression test has no guidance anywhere on any of this.
_failure scripts not handled in the template workflow or template Rakefile
The template workflow (run_inferno_execution_scripts.yml) and the template Rakefile.tt — what new test kits actually get — have no special logic for _failure scripts. A _failure script added to a test kit generated from the template will simply fail CI. The CI/CD doc says "all executed scripts are expected to complete successfully" with no caveat, so this gap is invisible to users.
Meanwhile, the inferno-core Rakefile passes allow_known_failures: true but the template's does not, meaning bundle exec rake execute_scripts:run_all behaves differently in inferno-core vs. a generated test kit — also undocumented.
There was a problem hiding this comment.
I intentionally put the failure handling into core, but not the test kits because
- In core, we need to be able to test the scripting infrastructure itself including catching failures.
- In test kits, a "failure" in the sense used here should never be expected. Failures are purple errors and misconfigured scripts.
Based on that, this complexity is not needed for test kits and so doesn't need to be documented here since this isn't supposed to be targeted at inferno-core developers. However, I will add a readme file to the execution_scripts directory in core that explains the failure handling there so that core developers can find that information.
There was a problem hiding this comment.
added a README.md file in core
3 participants
Summary
This PR documents the significant new features added in Core PR 760. Documenting how to use the features.
Updated pages include (links to localhost)
Testing Guidance
Render the content locally and check the rendered pages for